The present invention relates to a tool for testing and debugging computer programs, and more particularly to one that facilitates the testing of all relevant program code.
Computers and the software that causes them to perform useful work have become pervasive in virtually all aspects of daily life. Not only do they facilitate the performance of such mundane tasks as word processing, maintaining customer billing records and preparing personal income taxes, but they also are intimately involved in monitoring and regulating processes that can have an immediate impact on the well-being of personal property and even human lives, such as the operation of nuclear power plants, the safe coordination and control of airplane traffic and the delivery of vital medical services.
Therefore, one of the most critical phases in the development of computer software is the testing and debugging phase, in which faults or “bugs” in the software code are detected and corrected, so as to minimize the likelihood of the software failing to operate as intended. Bugs can be caused by the software developer's use of improper logic to implement the desired function, his/her failure to comply with the proper syntax dictated by the programming language in question, or even by something as trivial as a typographical error.
There are various tools that can facilitate the testing and debugging operation. Syntax errors are detected in the compilation phase. Certain known debugging tools permit the software developer to insert breakpoints at each location where the developer desires to examine the state of the program. When the execution of the program reaches this breakpoint, further execution is interrupted. The developer can then systematically examine the contents of the appropriate memory locations or registers where the program variables are stored, to determine if the program has returned the proper value upon execution of a particular instruction. Then, the developer can either step through the remainder of the program one instruction at a time or cause the program to resume execution until it arrives at another breakpoint, in either case observing the program state at various locations. Upon observing an improper value, the developer can analyze its significance and formulate the appropriate corrective action.
Alternatively, there are tools that permit the developer to insert tracepoints in lieu of breakpoints into the desired locations. Instead of causing the program to interrupt its operation, tracepoints initiate any one of several activities as defined by the software developer, such as generating an indicator confirming that the tracepoint has been reached, printing out data reflecting the state of the program at the time the tracepoint was reached, or collecting and recording such data for retrieval and analysis at a more convenient time. Such tools are particularly useful with programs in which an interruption of the program would negatively affect the program's behavior, e.g., programs that would interpret the interruption to be an indication of computer failure and take action to activate redundant or back-up systems.
The GNU Debugger (commonly referred to as GDB), a tool distributed by the Free Software Foundation, Inc. of Cambridge, Mass., is a commonly used debugging tool, especially for programs written in the C or C++ programming languages, and one that is amenable to use with either breakpoints or tracepoints. For example, Cygnus Solutions, Inc. of Sunnyvale, Calif. offers an extension to GDB named Introspect jointly developed by Cygnus Solutions and EMC Corporation of Hopkinton, Mass., assignee of the present invention), consisting of code inserted into GDB and a software “agent” running on the target program, that facilitates the definition and insertion of tracepoints into the target program.
However, the GDB, even in the extended version offered discussed above, does not by itself provide any guidance as to where to place such tracepoints in order to achieve optimal code coverage. In order to maximize the likelihood of finding latent bugs, thereby improving the overall quality of the computer program, it is important that adequate code coverage be achieved by the set of test cases being used to check the program. A particular set of test cases may not accomplish the desired results if critical areas of the code are not even exercised by the test cases. Over the years, various testing methodologies have been devised to measure the extent of code coverage produced by a particular set of test cases, determining the extent of such parameters as line coverage, statement coverage, branch coverage, path coverage; logical condition coverage, and loop coverage, to name a few. By knowing the extent of code coverage, the software developer can make an informed decision about where to add to or modify the test cases, so as to achieve the desired level of program quality or reliability. Generally, the developer often is required, for a variety of economic, technical or other reasons, to achieve a balance between the degree of code coverage (and consequently the acceptable degree of program quality) and the amount of time and resources available for performing the testing.
Even in those situations where the software developer wants to achieve maximum code coverage, the complexity of a large program may make it difficult to design test cases that will reliably ensure such coverage. Furthermore, simply knowing that maximum coverage has not been achieved may not adequately inform the developer how to revise the test cases to improve coverage. Rather, it is important for the developer to know exactly where all the gaps in coverage occur.
Modifications to an existing program continually evolve, due partly to the developer's desire to keep the program competitive by adding new or improved functionality, but due mostly to the detection of errors that, despite extensive testing during the development phase, only materialize after the software has been in daily use by hundreds of users in a real-life environment for an extended period of time.
Once a software developer has created a suite of software “fixes”, i.e., new software routines implementing new functionality and/or amended routines to correct previously detected errors, the question arises as how best to test these fixes so as to be satisfied that the new/corrected code has been adequately exercised, and that all intended results are achieved. Since it is quite likely that any new code would affect the preexisting code when integrated into the preexisting code, it has generally been customary to test the combination of the two, to uncover all potential problems. Nevertheless, it clearly would be preferable to eliminate as many bugs as possible from the fixes before they are integrated into the preexisting program, because eliminating those bugs after integration could be much more complex and could require much more time and many more resources. Intuitively, testing every single line of the code would appear most likely to yield optimum results. However, in most cases performing such extensive testing would be impractical, since it would occupy too many valuable resources, both in terms of personnel and testing resources, for too long a period of time.
Therefore, it is an objective of the present invention to improve the testing of computer program fixes by verifying adequate code coverage with minimal commitment of testing resources.
It is another objective of the present invention to achieve a testing procedure that automatically removes from the program code any testing code (e.g., tracepoints) that had been inserted into the program code to facilitate testing, after the execution of such testing code, so as to restore the program to its pre-testing condition and to eliminate undesirable interference with the behavior of the system on which the program code is designed to operate.
It is yet another objective of the present invention to achieve the above results in conjunction with an industry standard program debugger.
The above-stated objectives, as well as other objectives, features and advantages of the present invention will become readily apparent from the following detailed description, to be read in conjunction with the appended drawings.